Belgian fashion designer Raf Simons, considered one of the most talented of his generation, will leave Calvin Klein less than three years after joining it, parent company PVH said Friday.